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Abstract. We investigate the topological complexity of non Borel recognizable tree languages with 
regard to the difference hierarchy of analytic sets. We show that, for each integer n > 1, there is a 
D tlJ r l (Sj)-complete tree language C n accepted by a (non deterministic) Muller tree automaton. On 
the other hand, we prove that a tree language accepted by an unambiguous Biichi tree automaton 
must be Borel. Then we consider the game tree languages Wr LtK ), for Mostowski-Rabin indices 
(t, k). We prove that the D u ™ (Xj) -complete tree languages C n are Wadge reducible to the game 
tree language Wu tK ) for k — l > 2. In particular these languages W( LjK ) are not in any class _D Q (Sj) 
for a < 
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1. Introduction 

A way to study the complexity of languages of infinite words or infinite trees accepted by various kinds 
of automata is to study their topological complexity, and firstly to locate them with regard to the Borel 
and the projective hierarchies. It is well known that every w-language accepted by a deterministic Biichi 
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automaton is a n^-set. This implies that any w-language accepted by a deterministic Muller automaton 
is a boolean combination of Il^-sets hence a A^-set. l|Tho90[ ISta97l IPP04B . But then it follows from Mc 
Naughton's Theorem, that all regular u-languages, which are accepted by deterministic Muller automata, 
are also Ag-sets. The Borel hierarchy of regular cj-languages is then determined. Moreover Wagner 
determined a much more refined hierarchy on regular u-languages, which is in fact the trace of the 
Wadge hierarchy on regular u-languages, now called the Wagner hierarchy. 

On the other hand, many questions remain open about the topological complexity of regular languages 
of infinite trees. We know that they can be much more complex than regular sets of infinite words. 
Skurczynski proved that for every integer n > 1, there are some n°-complete and some ^-complete 
regular tree languages, [Sku93]. Notice that it is an open question to know whether there exist some 
regular sets of trees which are Borel sets of infinite rank. But there exist some regular sets of trees 
which are not Borel. Niwinski showed that there are some T, ^-complete regular sets of trees accepted 
by Biichi tree automata, and some II \ -complete regular sets of trees accepted by deterministic Muller 
tree automata, MNiw851 . Every set of trees accepted by a Biichi tree automaton is a X!^-set and every 
set of trees accepted by a deterministic Muller tree automaton is a n]^-set. Niwinski and Walukiewicz 
proved that a tree language which is accepted by a deterministic Muller tree automaton is either in the 
class II3 or n]^-complete, [NW03]. More recent results of Duparc and Murlak, on the Wadge hierarchy 
of recognizable tree languages, may be found in [Mur08 ( AD MN071 . 

It follows from the definition of acceptance by non deterministic Muller or Rabin automata and from 
Rabin's complementation Theorem that every regular set of trees is a A^-set, see BRab691 fPP04[ ITho90 , 
ILT94H . But there are only few known results on the complexity of non Borel regular tree languages. 
The second author gave examples of D^n (Sj)-complete regular tree languages in ||Sim92i Arnold and 
Niwinski showed in [AN08] that the game tree languages Wr b K -\ form a infinite hierarchy of non Borel 
regular sets of trees with regard to the Wadge reducibility. 

In this paper, we investigate the topological complexity of non Borel recognizable tree languages with 
regard to the difference hierarchy of analytic sets. We show that, for each integer n > 1, there is a 
D^n(X}) -complete tree language C n accepted by a (non deterministic) Muller tree automaton. On the 
other hand, we prove that non Borel recognizable tree languages accepted by Biichi tree automata have 
the maximum degree of ambiguity. In particular, a tree language recognized by an unambiguous Biichi 
tree automaton must be Borel. Then we consider the game tree languages Wu K \, for Mostowski-Rabin 
indices (t, k). We prove that the D^n (5]}) -complete tree languages C n are Wadge reducible to the game 
tree language Wn K \ for k — t > 2. In particular, these languages W( b K \ are not in any class D a (Y,\) for 
a < w w . 

The paper is organized as follows. In Section 2 we recall the notions of Biichi or Muller tree automata and 
of regular tree languages. The notions of topology, including the definition of the difference hierarchy 
of analytic sets, are recalled in Section 3. We show in Section 4 that there are D u n (S ^-complete tree 
languages C n accepted by Muller tree automata. We consider the complexity of game tree languages in 
Section 5. 
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2. Recognizable tree languages 

We recall now usual notations of formal language theory. 

When £ is a finite alphabet, a non-empty finite word over £ is any sequence x = a\ ■ ■ ■ a^, where G X 
for i = 1, . . . , k , and k is an integer > 1. The length of x is A;, denoted by \x\. The empfy word has no 
letter and is denoted by A; its length is 0. £* is the set of finite words (including the empty word) over S. 
Afinitary language over the alphabet £ is a subset of £*. 

The^rst infinite ordinal is cj. An w-worJ over £ is an -sequence ai ■ • • a n ■ ■ ■ , where for all integers 
i > 1, aj € £. When <r is an w-word over £, we write a = a(l)a(2) ■ ■ ■ a(n) • • • , where for all 
i, o~{i) € £, and a[n] = <t(1)(t(2) • • • cr(n) for all n > 1 and er[0] = A. 

The usual concatenation product of two finite words u and v is denoted u ■ v (and sometimes just uv ). 
This product is extended to the product of a finite word u and an a;- word v. the infinite word u ■ v is then 
the w-word such that: 

(u • v){k) = u{k) if k < \u\ , and (n • v)(k) = v(k — \u\) if k > |n|. 

The /?r<?/(x relation is denoted C: a finite word u is a prefix of a finite word v (respectively, an infinite 
word v), denoted u C v, if and only if there exists a finite word w (respectively, an infinite word w), such 
that v = u ■ w. 

The set of uj-words over the alphabet S is denoted by An oo-language over an alphabet S is a subset 
of 

We introduce now languages of infinite binary trees whose nodes are labelled in a finite alphabet S. 
A node of an infinite binary tree is represented by a finite word over the alphabet {/, r} where r means 
"right" and / means "left". Then an infinite binary tree whose nodes are labelled in £ is identified with a 
function t : {I, r}* — > S. The set of infinite binary trees labelled in £ will be denoted . 

Let t be a tree. A branch B of t is a subset of the set of nodes of t which is linearly ordered by the tree 
partial order C and which is closed under prefix relation, i.e. if x and y are nodes of t such that y € B 
and x Qy then x G B. 

A branch B of a tree is said to be maximal iff there is not any other branch of t which strictly contains 
B. 

Let t be an infinite binary tree in T^. If B is a maximal branch of t, then this branch is infinite. Let 
(ui)j>o be the enumeration of the nodes in B which is strictly increasing for the prefix order. 
The infinite sequence of labels of the nodes of such a maximal branch B, i.e. t(uo)t(ui) ■ ■ ■ t{u n ) • • • is 
called a path. It is an a;- word over the alphabet S. 

Let then L C S w be an ^-language over S. Then we denote 3Path(L) the set of infinite trees t in 
such that t has (at least) one path in L. 

We are now going to define tree automata and recognizable tree languages. 

Definition 2.1. A (nondeterministic topdown) tree automaton is a quadruple A = (K, S, A, qo), where 
K is a finite set of states, S is a finite input alphabet, qo G K is the initial state and A C K xY, x K x K 
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is the transition relation. The tree automaton A is said to be deterministic if the relation A is a functional 
one, i.e. if for each (g, a) G K x £ there is at most one pair of states (g', q") such that (g, a, g', q") G A. 
A run of the tree automaton A on an infinite binary tree t G T£ is a infinite binary tree p G such that: 

(a) p(X) = go and (b) for each u G {/, r}*, (p(u),t(u), p(u.l), p(u.r)) G A. 

Definition 2.2. A Biichi (nondeterministic topdown) tree automaton is a 5-tuple A = (K, S, A, go, F), 

where (K, X, A, go) is a tree automaton and F C K is the set of accepting states. 

A run p of the Biichi tree automaton A on an infinite binary tree t G is said to be accepting if for 

each path of p there is some accepting state appearing infinitely often on this path. 

The tree language L(A) accepted by the Biichi tree automaton A is the set of infinite binary trees t G 

such that there is (at least) one accepting run of A on t. 

Definition 2.3. A Muller (nondeterministic topdown) tree automaton is a 5-tuple A = (K, S, A, go, T), 

where (K, S, A, go) is a tree automaton and T C 2 K is the collection of designated state sets. 

A run p of the Muller tree automaton A on an infinite binary tree t G is said to be accepting if for 

each path p of p, the set of states appearing infinitely often on this path is in T . 

The tree language L{A) accepted by the Muller tree automaton A is the set of infinite binary trees t G 

such that there is (at least) one accepting run of A on t. 

The class REG of regular, or recognizable, tree languages is the class of tree languages accepted by 
some Muller automaton. 

Remark 2.4. Each tree language accepted by some (deterministic) Biichi automaton is also accepted by 
some (deterministic) Muller automaton. A tree language is accepted by a Muller tree automaton iff it 
is accepted by some Rabin tree automaton. We refer for instance to l|Tho901 IPP041 for the definition of 
Rabin tree automaton. 

Example 2.5. Let L C S w be a regular ^-language (see [PP04 ] about regular w-languages which are 
the w-languages accepted by Biichi or Muller automata). Then the set 3Path(L) C T£ is accepted by a 
Biichi tree automaton, hence also by a Muller tree automaton. 

The set of infinite binary trees t G having all their paths in L, denoted VPath(L), is accepted by a 
deterministic Muller tree automaton. It is in fact the complement of the set 3Path(S w — L). 

3. Topology 

We assume the reader to be familiar with basic notions of topology which may be found in IMo s801lLT941 
|Kec95l ISta97l IPPfJ4l . There is a natural metric on the set S w of infinite words over a finite alphabet S 
containing at least two letters which is called the prefix metric and defined as follows. For u, v G S w 
and u ^ v let 5(u, v) = 2~V cf ( u > i where l pre ^ u ^ is the first integer n such that the (n + l) st letter of 
u is different from the (n + l) s * letter of v. This metric induces on S w the usual Cantor topology for 
which open subsets of S w are in the form W ■ where W C £*. A set L C S w is a closed set iff its 
complement — L is an open set. 
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There is also a natural topology on the set T£ |Mos80, LT94, Kec95]. It is defined by the following 
distance. Let t and s be two distinct infinite trees in Tg. Then the distance between t and s is i where 
n is the smallest integer such that t(x) ^ s(x) for some word x G {I, r}* of length n. 
The open sets are then in the form To • T^ where To is a set of finite labelled trees. To • T^ is the set of 
infinite binary trees which extend some finite labelled binary tree to € To, to is here a sort of prefix, an 
"initial subtree" of a tree in to ■ T£. 

It is well known that the set T^, equipped with this topology, is homeomoiphic to the Cantor set 2^, 
hence also to the topological spaces where £ is an alphabet having at least two letters. 

We now define the Borel Hierarchy of subsets of It is defined similarly on the space T£ . 

Definition 3.1. For a non-null countable ordinal a, the classes E° and 11° of the Borel Hierarchy on 
the topological space £ w are defined as follows: 

E° is the class of open subsets of T, w , 11° is the class of closed subsets of 
and for any countable ordinal a > 2: 

E° is the class of countable unions of subsets of S w in \J 1<a ^y- 

11° is the class of countable intersections of subsets of in (J 7<a E°. 

For a countable ordinal a, a subset of Y, w is a Borel set of rank a iff it is in E° U 11° but not in 

U 7<Q (s°un°). 

There exists another hierarchy beyond the Borel hierarchy, which is called the projective hierarchy. The 
classes and 11*, for integers n > 1, of the projective hierarchy are obtained from the Borel hierarchy 
by successive applications of operations of projection and complementation. The first level of the pro- 
jective hierarchy is formed by the class 5]{ of analytic sets and the class n{ of co-analytic sets which 
are complements of analytic sets. In particular, the class of Borel subsets of is strictly included in the 
class I]{ of analytic sets which are obtained by projection of Borel sets. 

Definition 3.2. A subset A of is in the class 5] \ of analytic sets iff there exists another finite set Y 
and a Borel subset B of (X x Y) w such that x E A <-> 3y € Y w such that (x, y) G B, where (x, y) is 
the infinite word over the alphabet Ex7 such that (x, y){i) = (x(i),y(i)) for each integer i > 1. 

Remark 3.3. In the above definition we could take B in the class nP,. Moreover analytic subsets of S w 
are the projections of n°-subsets of T, w x uj w , where uj w is the Baire space, [Mos80|. 

We now define the notion of Wadge reducibility via the reduction by continuous functions. Let X, Y 
be two finite alphabets. For L C and L' C Y w , L is said to be Wadge reducible to L', denoted by 
L <w L', iff there exists a continuous function / : X w — > Y w , such that L = / _1 (L'). 

We now define completeness with regard to reduction by continuous functions. For a countable ordinal 
a > 1, and an integer n > 1, a set F C Y, w is said to be a S° (respectively, 11°, S^, Tlj^-complete set 
iff for any set E C Y w (with Y a finite alphabet): EeS° (respectively, E G 11°, T G S^, £ G 11^) 
iff E <\y F. 5]° (respectively n°)-complete sets, with n an integer > 1, are thoroughly characterized 
in llSta86l . 
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The Borel hierarchy and the projective hierarchy on are defined from open sets in the same manner 
as in the case of the topological space E^. 

The u;-language 7Z = (0* ■ 1) £J is a well known example of Il^-complete subset of {0, It is the set of 
w-words over {0, 1} having infinitely many occurrences of the letter 1. Its complement {0, 1} W — (0* • l) w 
is a Sf^-complete subset of {0, 

The set of infinite trees in Tg, where E = {0, 1}, having at least one path in the u;-language 1Z = (0* • l) w 
is E}-complete. Its complement is the set of trees in having all their paths in {0, 1} W — (0* • l) w ; it 
is II \ -complete. 

We now recall the notion of difference hierarchy of analytic sets. Let rj < u\ (where u\ is the first 
uncountable ordinal) be an ordinal and (Ao)g <v be an increasing sequence of subsets of some space X, 
then the set D n [(Ag)g <ri ] is the set of elements x G X such that x G Ag\\J gl<e Agi for some 9 < rj whose 
parity is opposite to that of r\. (Recall that a countable ordinal 7 is said to be even iff it can be written 
in the form 7 = a + n, where a is a limit ordinal and n is an even non-negative integer; otherwise the 
ordinal 7 is said to be odd; notice that all limit ordinals, like the ordinals u n , n > 1, or u) u , are even 
ordinals.) 

We can now define the class of ^-differences of analytic subsets of X, where X = T, w or X = . 
D V (H\) :={D v [(Ao)g <v ] I for each ordinal < r\ Ag is a Ej-set } 

It is well known that the hierarchy of differences of analytic sets is strict, i.e. that for all countable 
ordinals a < (3 < it holds that D a {T\) C Dp(E\). This is considered as a folklore result of 
descriptive set theory which follows from the existence of universal sets for each class D a (H\). Indeed 
we know first that the class Ej^ of analytic sets admits a universal set, see HKec951 page 205]or [Mos80 ( 
page 43]. Then, using classical methods of descriptive set theory, one can show that, for each countable 
ordinal a, the class Z) Q (E^) admits also a universal set, see HKan971 page 443]. This implies, as in the 
case of the Borel hierarchy in [Kec95 ( page 168], that the difference hierarchy of analytic sets is strict. 
As a universal set for the class D a (S]) is also a L> Q (E J) -complete set for reduction by continuous 
functions, this implies also that there exists a Z? a (E ^-complete set. 

Notice that in the sequel we shall only consider the classes D a (S\), for ordinals a < lo w , and that we 
shall reprove that there exists some D a (Ej)-complete subsets of T£ , giving examples which are regular 
sets of trees. 

Another folklore result of descriptive set theory is that the union |Ja<a) ^a(^i) represents only a small 
part of the class A3,. It is quoted for instance in [Ste82l or HKan971 page 443]. (It is noticed in [Ste82] 
that the union U a < Wl D a (S\) is strictly included in the class >4(nJ) which is the closure of the class 
Tl\ under Souslin's operation. The class ^l(nj) is included in the class A2 by IMos80 ( 2.B.5 page 75]). 
Notice however that this result is not necessary in the sequel. 
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4. D u n ( £ \ ) -complete recognizable languages 

It follows from the definition of the Biichi acceptance condition for infinite trees that each tree language 
recognized by a (non deterministic) Biichi tree automaton is an analytic set. 

Niwinski showed that some Biichi recognized tree languages are actually Sj^-complete sets. An example 
is any tree language T C in the form 3Path(L), where L C S w is a regular w-language which is 
a Il^-complete subset of In particular, the tree language C = 3Path(7*!.), where ft = (0* • l) w , is 
^-complete hence non Borel HNiw85l IPP041 |Sim92fl . 

Notice that its complement £~ = VPath({0, 1}^ — (0* • 1)^) is a Il}-complete set. It cannot be accepted 
by any Biichi tree automaton because it is not a 5^-set. On the other hand, it can be easily seen that it is 
accepted by a deterministic Muller tree automaton. 

The tree languages C and £~ have been used by the second author in |Sim92] to give examples of 
D w n(S}) -complete recognizable tree languages, for integers n > 1. We now give first the construction 
of a D Ld { -complete set. 

For a tree t G and u G {/, r}*, we shall denote t u : {I, r}* — > £ the subtree defined by t u (v) = t(u-v) 
for all v G {I, r}* . It is in fact the subtree of t which is rooted in u. 

Now we can define a D W (S}) -complete tree language C\. 

C\ = {t G T#, 1 | | 3n > tfn,. r G C and min{n > | tp. r G £} is odd }. 

Proposition 4.1. The tree language C\ is D w (S})-complete. 
Proof. We first show that the language C\ is in the class D^i^l)- 

Consider firstly, for some integer k > 0, the set T k = {t G Tf ^ | i^. r G £}. It is clear that this set 
is in the class S{ because the function Fj, : Tf Q ^ — > Tj^ ^ defined by Fj,(t) = tik. r is continuous and 

Tfc = F k ~ l (C) and the class S} is closed under inverses of continuous functions. 

Let now H n = {t G T^ ^ | 3k < n t[k. r G £}. This set is also in the class S| because the class is 

closed under finite (and even countable) union and H n = \J k<n T k . 

The sets H„ form an increasing sequence of S}-sets, and we can check that 

C-i = ^p n ) n<iJ ] 

We now prove that C\ is D^^^-complete. 

Let L C be a 1)^(5] J) -subset of E"-', where £ is an alphabet having at least two letters. Then there is 
an increasing sequence [A n ) n£uJ of ^-subsets of Y^ such that L = D^An)^^]. On the other hand, 
we know that the tree language C is S} -complete. Thus for each integer n > there exists a continuous 
function f n : Y" - T { w 01} such that A n = Z" 1 ^)- 

We now define a function F : S w -> T { w 1} by : for all a; G for all integers k > 0, F(x)(l k ) = and 
F(x)ik. r = fk(x). It is clear that the function i 7 is continuous because each function fj, is continuous. 
We can now check that for every x G Y^, x is in the set L = Du[{A n ) n< J\ iff there is an odd integer 
n such that x G A n \ [j k<n A k iff there is an odd integer n such that f n {x) G C and for all k < n 
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fk(x) G C~. 

This means that x G L = A^[(Ai)n<aj] iff F(x) G C\. 

Finally we have shown, using the reduction F, that L = Dw[{Arh)n<ui\ <w £\ and so the tree language 
C\ is Z) CJ (S})-complete. □ 

We can now generalize this construction to obtain some D W ^(S\) -complete tree languages, for every 
integer n > 1. 

Recall first that an ordinal a is strictly smaller than the ordinal uj n , where n > 2 is an integer, if and only 
if it admits a Cantor Normal Form 

a = u) n ~~ x ■ a n _i + uj n ~ 2 ■ a n „ 2 + • • • + u> ■ a\ + a 

where a n _i, a n _2, . . . , ao, are non-negative integers. In that case we shall denote Ord(a„_i, a n _2, ■ • • , ao) = 
LO n ~ x • a„-i + uj n ~ 2 ■ a„-2 + • • • + oj ■ a\ + a . 

Recall also that if a = Ord(a n _i, a n _2, • • • , ao) an d /? = Ord(6 n _i, & n -2, • • • , ^o)> then a < /3 if and 
only if there is an integer k such that < k < n — 1 and aj = bj for n — 1 > j > k and < b^. 

We now define the tree language C n , for n > 2, as the set of trees t G Tf Q ^ for which there exist some 
integers a n _i, a n _2, . . . ,oq > such that: 

1. tio, n - 1 . r . l a n _ 2 . r ...ia . r is in £ and the parity of Ord(a n _i, a n _2, • • • , ao) is odd, 

2. If Ord(6„_i, 6 n _ 2 , • • • , &o) < Ord(o„_i, a n _ 2 , • • • , ao) then the tree 
i | s„_i, r . | t„_ 2 j ... iio , r is not in £. 

Proposition 4.2. For each integer n > 2, the tree language £„ is D w n (S})-complete. 

Proof. The proof is a simple generalization of the proof of Proposition 14.11 Notice that we have to use 
the closure of the class S} under countable (and not only under finite) union. Details are here left to the 
reader. □ 



The tree languages C n can not be accepted by any Biichi tree automaton because each tree language 
accepted by a (non deterministic) Biichi tree automaton is an analytic set and D^n (S})-complete sets, 
for n > 1, are not in the class Sj. We are going to see that the tree languages C n are accepted by Muller 
tree automata. 

We now recall the following result proved by Niwinski in HNiw85l . see also for instance |PP04l [Tho90ll . 

Lemma 4.3. The language £~ = VPath({0, 1} W — (0*.l)") is a nj-complete set accepted by a deter- 
ministic Muller tree automaton. 

On the other hand, the tree language C is a X^-complete set. Thus it is not a n}-set otherwise it would 
be in the class A* = n n} which is the class of Borel sets by Suslin's Theorem. But every tree 
language which is recognizable by a deterministic Muller tree automaton is a II \ -set therefore the tree 
language C can not be accepted by any deterministic Muller tree automaton. However we can now state 
the following result. 
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Lemma 4.4. The language C is a ~E\ -complete set accepted by a non deterministic Biichi tree automaton, 
hence also by a non deterministic Muller tree automaton. 

Proof. We recall informally how we can define a non-deterministic Biichi tree automaton A accepting 
the language C. When reading a tree t G C, the automaton A, using the non determinism, guesses an 
infinite branch of the tree. Then the automaton checks, using the Biichi acceptance condition, that the 
sequence of labels of nodes on this branch forms an w-word in (0*.l) w , i.e. contains an infinite number 
of letters 1. □ 

Lemma 4.5. For each integer n > 1, the language C n is accepted by a (non deterministic) Muller tree 
automaton. 

Proof. We first construct a non deterministic Muller tree automaton A\ accepting the language C\. 

Recall that, for each tree t £ L\, there exists a least integer n > such that t\-n,. r £ C. This (odd) integer 
is defined in a unique way. One can now construct, from Muller tree automata A~ and A + accepting 
the tree languages C~ and C, a Muller tree automaton A\ accepting the tree language C\. Using the 
non-determinism, the automaton A\ will guess the (odd) integer n > and then, using the behaviour of 
A~ and A + , it will check that tin. r £ C and that, for every integer k < n, t^.j, C. 

We now give the exact construction of the non deterministic Muller tree automaton A\. 

Let E = {0, 1} and A~ = (K, S, A, qo, T) be a (deterministic) Muller tree automaton accepting the 

tree language Cr . 

And let A + = (K', S, A', q' ,!F') be a (non deterministic) Muller tree automaton accepting the tree lan- 
guage C. We assume that K !~l K' = 0. 

Then it is easy to see that the tree language L\ is accepted by the Muller tree automaton A\ = (K 1 , S, A 1 , 
q'q, JF 1 ), where 

K^KUK'uUlqlqf}, 

A 1 = AU A' U {(ql, a, ql,q ),(q{, a, q f ,q' ),(q f , a, q f ,q f ),(ql, a, q^,q ) \ a G {0,1}}, 
JF 1 =FUf'u{q f }. 

For every integer n > 1, we can construct in a similar way a Muller tree automaton A n accepting the 
tree language C n . 

Recall that for each tree t £ C n there exists a least ordinal a = Ord(a n _i, a n -2, ■ ■ ■ , ao) < u n such 
that t [ a n - 1 . r . l a n _2. r ... l a . r is in C. This (odd) ordinal is defined in a unique way. 

One can now construct, from the Muller tree automata A~ and A + accepting the tree languages £~ 
and C, a Muller tree automaton An accepting the tree language C n . Using the non-determinism, the 
automaton A n will guess the (odd) ordinal a = Ord(a n _i, a n _2, • • • ,ao) < to 11 and then, using the 
behaviour of A~ and A + , it will check that tj»„-i. r .i«n-2. r ...i«[). r is in C and that for each ordinal f3 = 
Ord(6 n _i, b n - 2 , ...,b ) < Ord(a n _i, a n _ 2 , . . . ,a ) the tree language i J 6„_ 1 . r . J 6„_ 2 r .. d b . r is not in C. 

□ 

We can now summarize the above results in the following theorem. 
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Theorem 4.6. For each integer n > 1, the language C n is a D w n(£})-complete set accepted by a (non 
deterministic) Muller tree automaton. 

Corollary 4.7. The class of tree languages recognized by Muller tree automata is not included into the 
boolean closure of the class of tree languages recognized by Biichi tree automata. 

Proof. We know that every tree language recognized by a Biichi tree automaton is a 5]{-set. But a tree 
language which is a boolean combination of S}-sets is in the class D u {Yl\) which does not contain all 
tree languages recognized by (non deterministic) Muller tree automata. □ 

Remark 4.8. We have given above examples of D^n -complete tree languages accepted by Muller 
tree automata. In a similar way it is easy to construct, for each ordinal a < lo^, a D Q (S}) -complete 
tree language accepted by a Muller tree automaton. Each ordinal a < lo^ may be written in the form 
a = Ord(a n _i, a n _2, • • ■ , ao) < w ™ for some integer n > 1 and where a n _i, a„_2, . . . , ao, are non- 
negative integers with a n _i ^ 0. 

The tree language T a is then the set of trees t G Tf Q ^ for which there exist some integers i , 6„_2 , • • • , bo > 
such that: 

1. Ord(6 n _i,6 n _ 2) • • • ,b ) < Ord(a„_i, a„_ 2 , . . . ,a ). 

2. t l b n _ 1>r l b n _ 2 t b is in C and the parity of Ord(6„_i, 6 n _2, . . . , bo) is odd iff the parity of 
Ord(a n _i, a n _ 2 , . . . , a ) is even. 

3. If Ord(c n _ 1 ,c n _2,... ,cq) < Ord(6„_i,6 n _ 2 , ■ ■ ■ ,M then the tree 

t; c n-i. r .; c n-2 ir ...;c . r is not in C 

The tree language T a is D a { 5] \) -complete and it is accepted by a (non deterministic) Muller tree au- 
tomaton. 

The above results show that the topological complexity of tree languages recognized by non deterministic 
Muller tree automata is much greater than that of tree languages accepted by deterministic Muller tree 
automata. 

Recall that a Biichi (respectively, Muller) tree automaton A, reading trees labelled in the alphabet S, is 
said to be unambiguous if and only if each tree t € admits at most one accepting run of A. 

A natural question is whether the tree languages C n could be accepted by unambiguous Muller tree 
automata. A first step would be to prove that the tree language C is accepted by an unambiguous Muller 
tree automaton. But this is not possible. We have learned by personal communication from Damian 
Niwinski that the language C is inherently ambiguous, HNiw0911 . 

We consider now the notion of ambiguity for Biichi tree automata and we shall prove in particular that a 
tree language accepted by an unambiguous Biichi tree automaton must be Borel. We shall indicate also 
why our methods do not work in the case of Muller automata. 
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We first recall some notations and a lemma proved in [FS03]. 

For two finite alphabets S and I.ifBCS^x X" and a e we denote B a = {f3 e X w \ (a, 0) e 
5} and PRO J S c (B) = {a£E u |B a / 0}. 

The cardinal of the continuum will be denoted by 2 N ° ; it is also the cardinal of every set S w or Tg, where 
£ is an alphabet having at least two letters. 

Lemma 4.9. ( llFS03l ) 

Let £ and X be two finite alphabets having at least two letters and B be a Borel subset of T, w x X u such 
that PROJ£"(-E>) is not a Borel subset of Then there are 2 H ° w-words a £ T, w such that the section 
B a has cardinality 2 N ° . 

Proof. Let £ and X be two finite alphabets having at least two letters and B be a Borel subset of T, w x X w 
such that PROJ s ^ (5) is not Borel. 

In a first step we prove that there are uncountably many a € T, w such that the section B a is uncountable. 

Recall that by a Theorem of Lusin and Novikov, see [Kec95, page 123], if for all a £ S", the section 
B a of the Borel set B was countable, then PRO Js^ (-B) would be a Borel subset of T, w . 

Thus there exists at least one a£S" such that B a is uncountable. In fact we have not only one a such 
that B a is uncountable. 

For q £ we have {a} x B a = B n [{a} x But {a} x X w is a closed hence Borel subset of 
S w x X w thus {a} x £? a is Borel as intersection of two Borel sets. 

If there was only one a S Y, w such that B a is uncountable, then C = {a} x £? Q would be Borel so 
D = B — C would be borel because the class of Borel sets is closed under boolean operations. 
But all sections of D would be countable thus PROJe"(-D) would be Borel by Lusin and Novikov's 
Theorem. Then PROJs^(-B) = {a} U PROJs"(-D) would be also Borel as union of two Borel sets, 
and this would lead to a contradiction. 

In a similar manner we can prove that the set [/ = {a € E u B a is uncountable } is uncountable, 
otherwise U = {ao,«i, • • • a n , . . .} would be Borel as the countable union of the closed sets {a{\, 
i > 0. 

For each n > the set {a n } x B an would be Borel, and C = U n&UJ {a n } x B an would be Borel as a 
countable union of Borel sets. So D = B — C would be borel too. 

But all sections of D would be countable thus PROJs"(-D) would be Borel by Lusin and Novikov's 
Theorem. Then PROJs" (B) = U U PRO Js^ (D) would be also Borel as union of two Borel sets, and 
this would lead to a contradiction. 

So we have proved that the set {a G S u B a is uncountable } is uncountable. 

On the other hand we know from another Theorem of Descriptive Set Theory that the set {a £ E w 
B a is countable } is a n;|-subset of see BKec951 page 123]. Thus its complement {a £ S w | 
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B a is uncountable } is analytic. But by Suslin's Theorem an analytic subset of S w is either countable or 
has cardinality 2 N °, MKec951 p. 88]. Therefore the set {a G S w | B a is uncountable } has cardinality 
2 K °. 

Recall now that we have already seen that, for each a G the set {a} x B a is Borel. Thus B a itself is 
Borel and by Suslin's Theorem B a is either countable or has cardinality 2^°. From this we deduce that 
{a G | B a is uncountable } = {a € T, u \ B a has cardinality 2 K °} has cardinality 2 Ko . □ 

This Lemma was used in [FS03] to prove that analytic but non Borel context-free w-languages have a 
maximum degree of ambiguity. 

Theorem 4.10. dlFSOll ) 

Let L(A) be a context-free a;-language accepted by a Biichi pushdown automaton A such that L(A) is an 
analytic but non Borel set. Then the set of w-words, which have 2 N ° accepting runs by A, has cardinality 

2 K _ 

Reasoning in a very similar way as in the proof of Theorem l4. lOl in IIFS03II . we can now state that analytic 
but non Borel tree languages accepted by Biichi tree automata have a maximum degree of ambiguity. 

If E is an alphabet having at least two letters, the topological space is homeomorphic to the topolog- 
ical space so we can first state Lemma l4~9l in the following equivalent form. 

Lemma 4.11. Let £ and K be two finite alphabets having at least two letters and B be a Borel subset of 
T£ x T% such that PROJt^C-B) is not a Borel subset of Tg. Then there are 2 H ° infinite trees t G T£ 
such that the section B t has cardinality 2 K ° . 

We can now state the following result. 

Theorem 4.12. Let L{A) C Tg be a regular tree language accepted by a Biichi tree automaton A such 
that L(A) is an analytic but non Borel set. Then the set of trees t G T£ which have 2 K ° accepting runs 
by A, has cardinality 2 N ° . 

Proof. Let A = (K, S, A, qo, F) be a Biichi tree automaton accepting a non Borel tree language L(A) C 
rg, and let R C x T% be defined by : 

R = {(t, p) | t G and p G T% is an accepting run of A on the tree t}. 

The set R can be seen as a tree language over the product alphabet S x K. Then it is easy to see that 
R is accepted by a deterministic Biichi tree automaton. But every tree language which is accepted by a 
deterministic Biichi tree automaton is a n^-set, see MMur0511 . Thus the tree language R is a n^-subset 
of the space T( Sx x)^ which is identified to the topological space x In particular, R is a Borel 
subset of Tg x Tj£. But by definition of i? it turns out that PRO J T - (R) = L(A). Thus PRO J T £ (R) is 
not Borel and Lemma l4. 1 ll implies that there are 2 K ° trees t G such that has cardinality 2 H °. This 
means that these trees have 2 K ° accepting runs by the Biichi tree automaton A. □ 
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Remark 4.13. The above proof is no longer valid if we replace "Biichi tree automaton" by "Muller 
tree automaton". Indeed if L(A) C is a regular tree language accepted by a Muller tree automaton 
A = (K, E, A, q ,F), then the set RCTg xl| defined by : 

R = {(t, p) \ t £ T£ and p G I# is an accepting run of A on the tree t}. 

is now accepted by a deterministic Muller tree automaton. Thus we can now only say that R is a Il];-set, 
and we cannot use the fact that R is Borel, which was crucial in the proof of Theorem 14. 121 

In particular, Theorem 14.1 21 implies the following important result. 

Corollary 4.14. Let L(A) C be a regular tree language accepted by an unambiguous Biichi tree 
automaton. Then the tree language L(A) is a Borel subset of T£. 

Remark 4.15. The result given by Corollary 14. 14l is weaker than the result given by Theorem 14. 121 This 
weaker result can be proved by a simpler argument. We give now this proof which is also interesting. 

Proof. Let L(A) C Tg be a regular tree language accepted by an unambiguous Biichi tree automaton 
A = (K, H,A,qo,F). Let R be defined as in the proof of Theorem l4.12l by: 

R = {(t, p) | t £ T£ and p G is an accepting run of A on the tree t}. 

The set R is accepted by a deterministic Biichi tree automaton so it is a n^-subset of the space Trs X K) u - 
Consider now the projection PROJ T - : T£ x T% -> Tg defined by PROJ T - (t, p) = t for all (t, p) G 
x Tji. This projection is a continuous function and it is injective on the Borel set R because the 
automaton A is unambiguous. By a Theorem of Lusin and Souslin, see [Kec95, Theorem 15.1 page 
89], the injective image of R by the continuous function PROJ^^ is then Borel. Thus the tree language 
L(A) = PROJ T £ (R) is a Borel subset of T£. ' □ 

Remark 4.16. The above result given by Corollary 14. 14l is of course false in the case of Muller automata 
because we already know an example of non Borel regular tree language accepted by a deterministic 
hence unambiguous Muller tree automaton. By Lemma 1431 the tree language C~ = VPath({0, 1} W — 
(0*.l) w ) is a II;} -complete set accepted by a deterministic Muller tree automaton. 

5. Game tree languages 

Game tree languages are particular recognizable tree languages which are defined by the use of parity 
games. So we now recall the definition of these games, as introduced in [AN08, ADMN07 ]. 

A parity game is a game with perfect information between two players named Eve and Adam, as in 
nAN08llADMN07l . 

The game is defined by a tuple G = (Vg, Vy, Move, po, rank). The sets Vg and Vy are disjoint sets of 
positions of Eve and Adam, respectively. We denote V = Vg U V\j the set of positions. The relation 
Move C V x V is the relation of possible moves. The initial position in a play is po G V. The ranking 
function is rank : V — > u and the number of values taken by this function is finite. 
At the beginning of a play there is a token at the initial position po where the play starts. The players 
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move the token according to the relation Move, always to a successor of the current position. The move 
is done by Eve if the current position is an element of V3, otherwise Adam moves the token. This way 
the two players form a path in the graph (V, Move). If at some moment a player cannot move then she 
or he looses. Otherwise the two players construct an infinite path in the graph, vq, v\, i?2, . . . In this case 
Eve wins the play if lim sup n ^ 00 rank(i> n ) is even, otherwise Adam wins the play. 
Eve (respectively, Adam) wins the game G if she (respectively, he) has a winning strategy. It is well 
known that parity games are determined, i. e., that one of the players has a winning strategy. Moreover 
any position is winning for one of the players and she or he has a positional strategy from this position, 
see MGTW021 for more details. 

We now recall the definition of game languages Wu K y 

A Mostowski-Rabin index is a pair (t, k), where 1 G {0, 1} and 1 < k < u. For such an index, we define 
the alphabet = {3, V} X {«.,... , k}. 

For a letter a G ^u, K ) we denote a = (a±, 02), where a% G {3, V} and a-i E {t, . . . , k}. 

For each tree t G we associate a parity game G(t) = (Vg, Vy, Move,po> rank), where 

. V 3 = {ve{l,r}*\t(v) 1 = 3}, 

• Vi = {v G {l,r}* I t(u)i = V}, 

• Move = {(w, wi) I w E {I, r}* and i G {/, r}}, 

• po = X is the root of the tree, 

• rank(-u) = t(v)2, for each v G {I, r}*. 

The set Q T£ ^ is the set of infinite binary trees t labelled in the alphabet X( tjK ) such that Eve 

wins the associated game G(t). 

The recognizable tree language Wn K \ is accepted by an alternating parity tree automaton of index (i, k). 
This notion will be useful in the sequel so we recall it now, as presented in [ADMN07]. 

Definition 5.1. An alternating parity tree automaton is a tuple A = (£, Q3, Qy, qo, 5, rank), where the 
set of states Q is partitioned in Q3 and Qy The set is the set of existential states and the set Qv is the 
set of universal states. The transition relation is 5 C Q x E x {/, r, A} x Q and rank : Q —> u is the rank 
function. A tree t G is accepted by the automaton A iff Eve has a winning strategy in the parity game 

(Qb x {l,r}*, Qv x {h r }*, (Qo, A), Move, fl), where Move = {((p,v), (q,vd)) \ v G dom(t), (p, t(v), 
d,q) G 5} and fl(q,v) = rank(g). 

Notice that it can be assumed without lost of generality that min rank(Q) is equal to or 1. The pair 
(min rank(Q), max rank(Q)) is called the Mostowski-Rabin index of the automaton. 
It follows from [Rab69] that any alternating parity tree automaton can be simulated by a non deterministic 
Muller automaton, see also IIGTW02I . 

There is a usual partial order on Mostowski-Rabin indices: (t, k) C (if, k') if either l' < 1 and k < k' 
(i.e. {l, . . . , k} C {l', k'}), or 1 = 0, 1? = 1, and k + 2 < k' (i.e. {1 + 2, . . . , k + 2} C {«,', . . . , «/}). 
The indices (1, n) and (0, n — 1) are called dual and (l, k) denotes the index dual to (i, k). 



Olivier Finkel, Pierre Simonnet / On Recognizable Tree Languages Beyond the Borel Hierarchy 



15 



It is easy to see that each tree language Wu K \ is accepted by an alternating parity tree automaton of 
index (t, k). 

Moreover the set Wt L>K \ is in some sense of the greatest possible topological complexity among tree 
languages accepted by alternating parity tree automata of index (l, k). This is expressed by the following 
lemma. 

Lemma 5.2. ( see BADMN07I ) 

If a set of trees T is recognized by an alternating parity tree automaton of index (l, k), then T <w W(^ K y 

In order to use this result to get a lower bound on the topological complexity of the game tree languages 
Wu K \, we first construct some alternating parity tree automata accepting the tree languages C and Cr 
defined in the preceding section. 

Lemma 5.3. The tree language C is accepted by an alternating parity tree automaton of index (1,2). 
Proof. Recall that C = 3Path(ft), where K = (0M) W . 

The tree language C is then accepted by the alternating parity tree automaton A = (T,,Q^,Q\/,qo,S, rank), 

where 

£ = {0,1}, 

Qb = Q = {<?o,<7i}, 

Qv = 0, 

5 = {(q,l,d,qi), (q,0,d,q ) | q G Q and d £ {l,r}}, 

rank(go) = 1 and rank(gi) = 2. □ 
Notice that in the above automaton A all states are existential. 

Lemma 5.4. The tree language Cr is accepted by an alternating parity tree automaton of index (0, 1). 
Proof. Recall that Cr = - C = VPath({0, 1} U - (0*.l) w ). 

The tree language £~ is then accepted by the alternating parity tree automaton A' = (S, Q' 3 , Q'y, qQ,5', rank'), 
where 

£ = {o,i}, 
Q' 3 = 0, 

Qv = Q' = Wo,<fi}> 

5' = {(q', 1, d, q' x ), (q', 0, d, q' ) \ q' G Q' and d G {/, r}}, 
rank'^) = and rank'(q^) = 1. 

□ 

Notice that in the above automaton A' all states are universal. 

Remark 5.5. The I] {-complete tree language £ is accepted by an alternating parity tree automaton of 
index (1, 2) and the n{-complete tree language Cr is accepted by an alternating parity tree automaton of 
index (0, 1). In fact for every tree language T accepted by an alternating parity tree automaton of index 
(1, 2) (respectively, (0, 1)) it holds that T is in the class T,{ (respectively, n{), see RADMN071 Theorem 
3.6]. 
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Recall now the definition of the ^-complete tree language C\. 

C\ = {t G Tf Q j-, | 3n > tin. r G £ and min{n > | t\n. r G £} is odd }. 

We can now state the following result. 

Lemma 5.6. The tree language C\ is accepted by an alternating parity tree automaton of index (0, 2). 

Proof. Let, as in the proofs of the two previous lemmas, A = (£, Q 3 , Q\/, qo, 5, rank) be an alternating 
parity tree automaton of index (1,2) accepting the tree language C = 3Path(7£), and A' = (E, Q' 3 , 
QL, Qq, 5' , rank') be an alternating parity tree automaton of index (0, 1) accepting the tree language C~ . 
We assume that Q n Q' = 0, where Q = Q 3 U Qy = Q 3 and Q' = Q' 3 U Q' y = Q' v . 
It is then easy to see that the tree language C\ is accepted by the alternating parity tree automaton 
.4 1 = (Z,Q 3 , Q\,ql,5 x , rank 1 ), where 

£ = {o,i}, 

Q\ = Q3 U Q' 3 U {q 3 } = Q 3 U {q 3 }, 

Q^ = QvUQ'yU{q 1 ,q 1 1 } = Q' v U{q 1 ,q 1 1 }, 

S 1 = 5 U 5' U {(ql,a,l,q 3 ),(q^,a,r,q' ),(q 3 ,a,r,q ),(q 3 ,a,X,ql),(ql,a,r,q' ),(ql,a,l,ql) \ a G 
{0,1}}, 

rank 1 (q) = rank(g) for q G Q, 
rank 1 (g / ) = rank'(g') for q' G Q 1 , 
rank x (^) = 0, rank^gj) = 1. 

□ 

Notice that in the above construction of the alternating automaton A 1 the universal states q^ , q\ and the 
existential state q 3 are used to choose, when reading a tree t G C\, the least integer n such that tp. r G C 
and to check that this integer is really the least (and odd) one with this property. 

In a very similar manner, for each integer n > 1, we can define an alternating parity tree automaton A n 
of index (0, 2) accepting the language C n . The complete description would be tedious but the idea is 
that now the additional universal or existential states not in Q U Q' are used to choose, for a given tree 
t G L n , the least ordinal a = a; n_1 • a n _i + cu n ~ 2 ■ a n _2 + • . . + u> ■ a\ + ao such that t\a n — l. r .l a n — 2 . r .../ao -r 
is in C and to check that a is odd and that for any smaller ordinal f3 = Ord(6 n _i, 6 n -2, • • • , &o) < a, the 
tree t^,^^^.^^ is not in C. 

We can then state the following result. 

Proposition 5.7. For each integer n > 1, the tree language C n is accepted by an alternating parity tree 
automaton of index (0, 2). 

We can now infer from Theorem I4.6L Proposition 15 .7 1 and Lemma [5721 the following result. 

Theorem 5.8. For each integer n > 1, the D u n (X \) -complete tree language C n is Wadge reducible to 
the game tree language W( 0) 2)> i-e. C n <w W( 0>2 y In particular the language WV2) is not in any class 
£> a (E|) for a < uj w . 
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On the other hand, Arnold and Niwinski proved in [ AN08 ] that the game tree languages form a hierarchy 
with regard to the Wadge reducibility. 

Theorem 5.9. (HANOI) 

For all Mostowski-Rabin indices {i, n) and (</, «/), it holds that : 

(i, k) E {if, «') if and only if W M < w Wy^i) 
Then we can state the following result. 

Theorem 5.10. For each integer n > 1 and each Mostowski-Rabin index (t, k) such that (0, 2) jZ (t, «) 
or (t, k) = (1, 3) = (0, 2), the D u n (£*)-complete tree language C n is Wadge reducible to the game tree 
language Wu K y i.e. C n <w Wn K y In particular the language Wu K \ is not in any class D a (S\) for 
a < uj^. 

Proof. The result follows directly from Theorems 15.81 and |5T9l in the case (0, 2) jZ (t, k). What remains is 
the case of the index (1, 3) which is the dual of the index (0, 2). But it is proved in HAN08I Lemma 1] that 
Wj^r coincide with W( tl/e ) = T£ } — up to renaming of symbols. On the other hand, we know 

from Theorem 15.81 that for each integer n > 1, the -D^n+i (S^)-complete tree language C n+ \ is Wadge 
reducible to the game tree language Wiq^), i- e - Ai+i <w ^(0,2) • This is easily seen to be equivalent 
to Tj£ 1} - £ n+ i < w PF(o, 2 ), i-e. T^ ^ - £ n+i < w ^(1,3)- But £n is 5 W » (Sj) -complete and C n+1 
is D^n+i (S})-complete so it follows from the properties of the difference hierarchy of analytic sets that 
£-n <w Tf Q x -, — C n+ \ and so C n <w W(i,3) by transitivity of the relation <w- □ 



6. Concluding remarks 

We have got some new results on the topological complexity of non Borel recognizable tree languages 
with regard to the difference hierarchy of analytic sets. In particular, we have showed that the game tree 
language W(o,2) i s not i n anv class D a {T,\) for a < lo^. The great challenge in the study of the topo- 
logical complexity of recognizable tree languages is to determine the Wadge hierarchy of tree languages 
accepted by non deterministic Muller or Rabin tree automata. Notice that the case of deterministic Muller 
or Rabin tree automata have been solved recently by Murlak, [Mur08|. 

It would be interesting to locate in a more precise way the game tree languages with regard to the 
difference hierarchy of analytic sets. We already know that 1^(0.2) i s not i n an Y class D a (S}) for a < lu^. 
Is there an ordinal a such that W(p t 2) i s i n D a (Y\) and then what is the smallest such ordinal a? The 
same question may be asked for the other game tree languages Wt i K y On the other hand, there are some 
sets in the class A<j, which does not belong to the d-algebra generated by the analytic sets, see MKec95[ 
Exercise 37.8]. Could we expect that W(o,2) or another game tree language Wt hK \ is such an example? 

Acknowledgements. We thank the anonymous referees for their very helpful comments which have led 
to a great improvement of our paper. 
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